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Abstract 

To examine the impact of transitioning 4th grade reading comprehension assessments 
to the computer, 219 fourth graders were randomly assigned to take a one-hour reading 
comprehension assessment on paper, on a computer using scrolling text to navigate 
through passages, or on a computer using paging text to navigate through passages. This 
study examined whether presentation form affected student test scores. Students also 
completed a computer skills performance assessment, a paper based computer literacy 
assessment, and a computer use survey. Results from the reading comprehension assess- 
ment and the three computer instruments were used to examine differences in students 
test scores while taking into account their computer skills. ANOVA and regression anal- 
yses provide evidence of the following findings: 

1. There were no significant differences in reading comprehension scores across 
testing modes. On average, students in the paper group (n=75) answered 58.1% 
of the items correctly, students in the scrolling group (n-70) answered 52.2% 
of the items correctly, and students in the whole page group (n-74) answered 
56.9% of the items correctly. The almost a 6% point difference in scores between 
the paper and scrolling groups was not significant at the p<.05 or p<.l level. 
Although the results suggest that, across all students, the modal effect is not 
statistically significant, this finding may be due in part to the unusually high 
computer access and higher socio-economic status of the sample. 

2. There were no statistically significant differences in reading comprehension 
scores based on computer fluidity and computer literacy, but a pattern in per- 
formance suggests that students are disadvantaged by the scrolling text mode, 
particularly students with lower computer .sk ill s. 

3. The majority of students who took the reading test on a computer indicated that 
they would prefer to take the test on computer. Although this sample did not 
include many students who had limited prior computer experience, the survey 
responses, completion rates, and student observations provide evidence that 
computer anxiety generally did not interfere with students’ ability to take the 
assessment. 

4. Providing highlighters and review markers is useful for some students. 

The results of this study suggest that further research is warranted to understand differ- 
ences in scores when reading comprehension assessments are administered via computer 
to a larger and more diverse group of students. 
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Background/Purpose 

The No Child Left Behind legislation requires that all students in grades 
3-8 be tested each year in English Language Arts and mathematics. For 
many states, complying with this new law requires a significant increase in 
the amount of student testing conducted each year. This proliferation of 
testing without an associated increase in funding to pay for the new tests 
has caused many state assessment divisions to examine new ways to make 
the assessment process more efficient. One alternative that several states 
have considered is computer-based delivery of assessments. According to 
Education Week, 12 states and the District of Columbia were planning to 
administer some form of computer-based state exams during the 2002- 
2003 school year (Editors, 2003). Although computer-based test admin- 
istration has the potential to save time and money while also returning 
results in a more timely manner, research must be undertaken to under- 
stand how changes in testing mode affect student performance. 

To date, a substantial body of research has examined the comparability 
of scores provided by paper-based and computer-based tests (Bunderson, 
Inouye, 8c Olsen, 1989; Bangert-Downs 1993). The majority of this 
research, however, has focused on young adults and adult populations. In 
addition, few existing studies have focused specifically on reading compre- 
hension tests, particularly for young readers. 

When reading comprehension tests that require examinees to read 
extended passages are presented on a computer, test developers are 
faced with an important design decision: how to allow students to navi- 
gate through a passage. Currently, one of two options are typically used: 
a) the passage is broken into subsections and each subsection is presented 
as a single block of text; or b) the passage is presented as a single block 
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of text through which the examinee must scroll. While students who are 
accustomed to using the Internet or reading text passages on a computer 
may find scrolling through a reading passage to he a natural process, those 
students who are less familiar with technology may find scrolling to be a 
relatively novel experience and in turn may be negatively affected by this 
format. If the students’ ability to scroll through a reading passage does 
affect their performance on a reading comprehension assessment, the 
computer-based version of this assessment introduces a new construct, 
namely computer literacy. If computer literacy is not part of the construct 
being measured by a reading comprehension test, then this “ancillary 
non-construct element” (Haertel & Wiley, 2003, 1) should be eliminated 
or reduced as much as possible from the reading comprehension assess- 
ment. In other words, the mode of assessment administration should not 
unfairly bias the scores of students based on their computer literacy skills. 
As Pommerich and Burden (2000) highlight, “because of subtle differences 
in test presentation across administration modes, care must be taken to 
ensure that examinees respond to item content only” (p. 1). 

Research on the impact of transitioning passage-based tests from 
paper to computers has been conducted for almost two decades. In 1986 
Haas and Hayes found that students taking tests on computer where the 
reading passages were more than one page in length achieved lower test 
scores. Mazzeo and Harvey (1988) also found that passage-based tests 
administered on computer may be more difficult than paper-based tests. 
This increased difficulty may be due to students being uncomfortable with 
navigating through a reading passage on computer. However, as the student 
to computer ratio decreases in schools and students are given more access 
to computers outside of school, some experts predict that the modal effect 
will become less of an issue. As an example, Clariana and Wallace (2002, 
p. 599) write, “As students become as familiar with computer-based testing 
as they are with paper-based testing, the test mode effect should decrease 
or disappear.” Until that time, however, it is important for test designers 
to develop a better understanding of the extent to which modal effects 
exist for different populations and to develop strategies that reduce the 
effect for these populations. 

To this end, in 1998 and2000 Pommerich conductedresearch on English, 
Reading, and Science passage-based tests using more than 20,000 11'^’^ and 
12*^^ graders (Pommerich, 2004). In the 2000 administration, students 
were randomly assigned to take the test on paper, on computer using 
scrolling text, or on the computer using paging text. Although students 
who took the English and Science test on computer scored significantly 
higher than students who took these tests on paper, there were no signifi- 
cant differences in scores across modes for the Reading test. Pommerich 
also notes that while significant differences were found in the English and 
Science tests, adjustments for multiple comparisons were not made and 
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“the effect sizes for the mean differences observed across hoth studies 
were no larger than +/- 0.15 in any content area, which is considered small 
hy Cohen’s (1988) standard” (Pommerich, 2004, 20). At an item level, 
Pommerich found some significant differences and concluded that “exam- 
inees responded differently to some items under different (presentation) 
variations” (p. 37). 

In 2002, Choi and Tinkler presented findings from a comparability 
study that examined differences in performance for 3 ^^ and 10*^^ grade stu- 
dents on the Oregon state math and reading tests administered on paper 
and computer. Using a counterbalanced design, all students were adminis- 
tered half of the items in each subject area on paper and half of the items on 
computer. The largest difference in test scores was found on the 3 ^'^ grade 
reading test. The authors concluded that “scrolling reading passages on 
computer screens seems to interfere with students’ test-taking behavior, 
especially the test-taking behaviors of younger students. Providing page- 
up and page-down buttons in place of a vertical scroll bar may alleviate the 
interference that might have affected younger students” (Choi & Tinkler, 
2002 , 10 ). 

The purpose of the study presented here is to examine differences in 
performance when two different computer-based test formats and a tra- 
ditional paper-and-pencil based format were used to present reading pas- 
sages to 4* grade students. Specifically, the research questions are: 

1. When 4* grade reading comprehension assessments are 
transitioned to computers, what is the impact on students’ 
test scores? 

2. Are there differences in test scores when reading passages are 
presented on the computer using scrolling text versus paging 
text? 

3. How is student performance (modal effect) related to their 
computer skills, computer literacy, and prior computer use? 

Results from this research provide evidence of whether computer 
presentation, with scrolling or whole page text format, interferes with 
measuring reading comprehension skills of fourth grade students. This 
research was federally funded through the Enhancing State Assessment 
grant program and conducted collaboratively with Vermont, Rhode Island, 
New Hampshire, Maine, the Education Development Center (EDC), and 
CAST. 
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Design 

To examine whether and how the presentation of reading passages 
affected the test performance of 4* grade students, 219 students from eight 
Vermont schools were randomly assigned to perform the same reading test 
in one of three modes: 1) on paper, 2) on computer with scrolling reading 
passages; and 3) on computer with passages divided into sections that were 
presented as “whole pages” of text. The participating schools were selected 
with the cooperation of the state Director of Assessment. When selecting 
schools, we aimed to acquire a mix of rural, suburban, and urban schools 
and a mix of students that included English Language Learners (ELL), 
students identified with special needs in the area of language/reading, as 
well as non-special education native English-speaking students. Since ELL 
students tended to be clustered within schools, and since the location of 
the school could not be manipulated, random assignment occurred within 
rather than across schools. Through this three group randomized design, 
the research compares the effect of using paper and pencil versus text pre- 
sented using scrolling or whole page format on student test performance. 

To control for effects that might result from differences in the com- 
puters available within each school, the research team brought into each 
school a set of Macintosh 12-inch iBooks (laptops) with traditional hand- 
held mice. All students performing the reading test on computer used one 
of the research team’s laptop computers. In addition to performing the 
same four-passage reading test, all students also completed a computer 
fluidity test, a computer literacy test, and a computer use survey. The 
computer fluidity test was administered to all students on a laptop. The 
computer literacy and the computer use surveys were administered to all 
students on paper. The purpose of administering these three additional 
instruments was to collect multiple measures of students’ computer skills, 
knowledge, and use so that we could examine the extent to which any 
modal affects are related to differences in students’ ability or familiarity 
with using a computer - constructs that are not intended to be measured 
by the reading comprehension test. 
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Data Collection 

Data was collected from students in the eight Vermont schools during 
January 2004. Students were given one hour to complete the reading com- 
prehension assessment and an additional hour to complete the computer 
fluidity, literacy, and use instruments. The number of grade students 
per school ranged from 13 to 45. Table 1 summarizes demographic infor- 
mation about students participating in the study. Specifically, Table 1 indi- 
cates that 99 students (45%) attended rural schools, 83 (38%) attended 
suburban schools, and 37 (17%) attended urban schools. In addition, 54 
students (25%) reported that they spoke a language other than English at 
home. 

Table 1 : Demographic Summary for Participating Students 


Geography 

Number 

of 

Students 

Rural 

99 

Suburban 

83 

Urban 

37 

Multilingual Status 


Multilingual 

54 

Non multilingual 

165 


Instruments 

Students participating in this study completed four data collection 
instruments in the following order: 1) Reading Comprehension Test; 
2) Computer Fluidity Test; 3) Computer Literacy Test; and 4) Computer 
Use Survey. Below, each instrument is described in greater detail. In addi- 
tion, we present information on the reliability and validity of each scale. 

Reading Comprehension Test 

One test form, containing 4 reading passages and 19 associated 
multiple choice items was administered to all students participating in 
this study. Passages and multiple choice items were taken directly from 
released National Assessment of Educational Progress (NAEP), Progress 
in International Reading Literacy Study (PIRLS), and New Hampshire 
state assessments. Passages and items were chosen based on their grade 
level appropriateness and the item characteristics. Specifically, for the vast 
majority of items, the item difficulty was such that between 40% and 60% 
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of students had answered the item correctly during the item’s original 
administration. Given that these items were of moderate difficulty, we 
anticipated that they would he most sensitive to modal effects. 

Both computer-based assessments began by requiring students to 
enter an identification number, which was pre-assigned to students based 
on their mode of delivery. The program then led the students through a 2 
minute and 20 second tutorial. Based on findings from a small beta study, 
the tutorial was changed from being interactive to being passive in order to 
encourage students to read the instructions and not simply speed through 
the tutorial without understanding the functionality of the test delivery 
interface. Students were told that the computer would show them how to 
take the assessment and that they did not need to press any buttons during 
the tutorial. The passive tutorial showed students how to select answers, 
read the passage, highlight text, and mark answers for review without 
requiring them to practice the tasks. Figure 1 displays a screen shot of the 
test interface for the scrolling passage and the whole page formats. The 
computer-based versions of the test may also be accessed at http://www. 
intasc.org/studies/enhanced_assessment/description.shtml . 
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Figure 1: 


Computer-based Reading Test Interfaces 

Scrolling 



Whole Page 
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The passages and multiple-choice items for the paper version were 
presented on double-sided pages and the number of words on each line, 
number of lines per page, and text wrap were identical to the whole page 
computer-based format. The font size and style were nearly identical across 
all three test forms in order to minimize differences that result from minor 
changes in the presentation of passages and items. As mentioned earlier, 
all students who performed the computer versions of the test used the 
same type of 12-inch laptop computer so that we could control for differ- 
ences in the size of text that may result from different screen sizes and/or 
resolution. 

Computer Fluidity Test 

After completing the reading comprehension assessment, all students 
were asked to complete a computer fluidity test that consisted of four sets 
of exercises. The purpose of the computer fluidity test was to measure 
students’ ability to use the mouse and keyboard to perform operations 
similar to those they might perform on a test administered on a computer. 
In this report, we refer to these basic mouse and keyboard manipulation 
skills as “computer fluidity.” 

The first exercise focused on students’ keyboarding skills. For this 
exercise, students were allowed two minutes to keyboard words from a 
given passage. The passage was presented on the left side of the screen and 
students were required to type the passage into a blank text box located 
on the right side of the screen. The total number of characters that the 
student typed in the two-minute time frame was recorded. 

After completing the keyboarding exercise, students performed a set 
of three items designed to measure students’ ability to use the mouse to 
click on a specific object. For these items, students were asked to click on 
hot air balloons that were moving across the computer screen. In each set 
of hot air balloons, the amount of time and the number of times the mouse 
button was clicked while clearing the screen were recorded. 

The third computer fluidity exercise measured students’ ability to use 
the mouse to move objects on the screen. For this exercise, students were 
presented with three items each of which asked students to drag objects 
from the left hand side of the screen to a target on the right hand side of 
the screen. Similar to the clicking exercise, for each item the amount of 
time and the number of times the mouse was clicked were recorded. 

Finally, the fourth exercise was designed to measure how well students 
were able to use the keyboard’s arrow keys to navigate on the screen. For 
this exercise, students were asked to move a ball through a maze by using 
the arrow keys. Students were shown where on the keyboard to And the 
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arrow keys. The time required to reach the end of the maze were recorded. 
As described in the analysis section, the data from the keyboarding, 
clicking, drag and drop, and arrow key exercises was combined into a single 
scale to produce a computer fluidity score for each student. 

Computer Literacy Test 

After finishing the computer fluidity exercises, students were asked to 
complete a short paper-based computer literacy test. The purpose of this 
test was to measure students’ familiarity with computing terms and func- 
tionality. Virginia and North Carolina have administered multiple -choice 
computer literacy tests to students at lower grade levels. Eleven released 
multiple -choice items from previously administered VA and NC assess- 
ments were used in the computer literacy test as part of this research. 
Items were chosen based on their alignment with the International Society 
for Technology in Education standards. 

Computer Use Survey 

Lastly, students were asked to complete a paper-based survey. This 
survey was adapted from the grade five student survey constructed for the 
Use, Support, and Evaluation of Instructional Technology (USEIT) study 
(see Russell, Bebell, and O’Dwyer, 2003). Students were asked questions 
focusing on their specific uses of technology in school and at home, their 
comfort level with technology, as well as some demographic information. 
Students who took the assessment on laptops were asked four additional 
open-ended questions about their experience taking the assessment on a 
computer. 

Scale Development 

As described above, three instruments were administered to students 
in order to measure their computer fluidity, literacy, and use. Each of 
these instruments was developed specifically for this study. While items 
that comprised the literacy and use instruments were taken directly from 
instruments that have been used in previous research and/or state test 
administrations, the specific set of items that comprise each instrument 
had not previously been used in tact. In addition, the items that formed the 
computer fluidity test were developed by the research team and had not 
previously been administered to a large number of students. Thus, before 
information from these three instruments could be used for analytic pur- 
poses, scale scores had to be developed and the reliability of these scales 
was examined. To this end, two sets of analyses were conducted to create 
and then examine the reliability of these scales. Eirst, principal compo- 
nent analyses were performed on each instrument to examine the extent 
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to which the items could he grouped to form a single score. In cases where 
all items could not he combined to form a single scale, principal compo- 
nent analyses were used to identify a subset of items that formed a unidi- 
mensional scale. Scale scores were then created for each student. Second, 
Cronbach’s alpha was calculated for each scale to examine the reliability of 
the scale. In cases where the scale had unacceptably low reliability (below 
.60), item to total score correlations were examined to identify items that 
were contributing to low reliability. These items were then dropped from 
the scale, new scale scores were created, and the reliability analysis was 
repeated. 

The final computer fluidity scale was created through a factor analysis 
of the weighted click variable, the drag times, the weighted arrow time, 
and weighted reversed keyboarding times. This yielded a one-factor solu- 
tion which accounts for 45.1% of the total variance in fluidity scores with 
an alpha reliability of 0.76. Factor loadings of each variable used range 
from 0.46-0.78. The computer literacy scale was created from five items, 
whose content was based on word processing skills and computer termi- 
nology. When a principal component factor analysis was run on these five 
items, a one-factor solution that accounted for 38.4% of the variance and 
had an alpha reliability coefficient of 0.60 was achieved. Factor loadings on 
the five items ranged from 0.57-0.74. The home computer use scale was 
created from seven of the eight home computer use survey items. A prin- 
cipal components factor analysis of these seven items yielded a one factor 
solution that accounted for 49.7% of the variance and had an alpha reli- 
ability of 0.83. Factor loadings on the seven items ranged from 0.53-0.80. 
Lastly, a school computer use scale was created from four of the survey 
items which focus on writing, multimedia, and research. A principal com- 
ponents factor analysis yielded a one factor solution that accounted for 
50.8% of the variance and had an alpha reliability of 0.68. Factor loadings 
on the four items ranged from 0.68-0.74. 


Results 

To examine the extent to which the mode of testing and the way in 
which extended text passages presented on a computer affect student per- 
formance, a series of analyses were performed. These analyses include a 
comparison of completion rates and a comparison of mean percent of cor- 
rect responses across the three modes. To examine the extent to which 
prior experience and skill using a computer interacted with the presenta- 
tion mode, analyses were conducted using the computer fluidity, literacy, 
and use measures. For each of these measures, students were divided 
into three groups representing high, medium, and low levels of fluidity, 
literacy, or use. The modal effect was then examined within each group 
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by comparing performance across the three presentation modes. In addi- 
tion, regression analyses were conducted to examine the modal effect 
after adjusting for differences in prior fluidity, literacy, and use. Finally, to 
examine the extent to which the modal effect differed between boys and 
girls or between lEP and Non-IEP students, analyses were performed by 
gender and lEP status. Eindings from each of these analyses are presented 
below. 

Completion Rate 

Of the 219 participating students, nine students did not reach the 
last item on the test in the given one hour timeframe. Three of the stu- 
dents who did not reach the last item were administered the assessment 
on paper, three were administered the assessment on laptop with scrolling 
text, and three were administered the assessment on laptop with whole 
page text. Some students also skipped items while taking the test. A total 
of 194 students (88.6%) answered all 19 items. Of the 25 students who left 
at least one item unanswered, 14 students were administered the assess- 
ment on paper, 5 students were administered the assessment on laptop 
using scrolling text, and 6 students were administered the assessment on 
laptop using whole page text. Three of the paper students who left items 
unanswered did not answer items that were presented on the back of at 
least one page. It also is noteworthy that students who took the assess- 
ment on laptops were prompted at the end of the assessment with a sum- 
mary of the items that they left unanswered. This automatic prompt may 
have led to a lower number of students leaving items unanswered. 

Reading Comprehension Percent Correct 

The average student score on the reading comprehension test was 
55.8%. The lowest score was 10.5%, the highest score was 100%, and the 
standard deviation of scores was 23.4. If one of the testing modes unfairly 
advantaged students, we would expect a significant difference in scores 
across modes of delivery. As seen in Table 2, there is nearly a six point dif- 
ference in the mean score between the paper and scrolling groups. A one- 
way Analysis of Variance (AN OVA), however, indicates that the differences 
among groups were not statistically significant (p = .278). 

Table 2: Comparison of Scores Among the Three Presentation Modes 


Group 

N 

Mean 

Standard 

Deviation 

F 

DF 

P 

Paper 

75 

58.1% 

.243 

1.289 

218 

.278 

Whole Page 

74 

56.9% 

.227 




Scrolling 

70 

52.2% 

.230 
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The box-plot in Figure 2 displays the differences in test scores across the 
three presentation inodes. Although there is a visible difference between 
the median for the scrolling condition and the two other presentation 
modes, this score difference is relatively small compared to the standard 
deviation for each presentation mode. 

Figure 2: Box-plot Depicting Score Differences Among the Three Presentation 

Modes 


100 



dJ 

Q_ 


20 - 


0 -^ 

paper scroll whole page 


N=75 N=70 N=74 


Maximum score 


75th percentile 


Median 


25th percentile 


Minimum score 


To examine whether the presentation mode may have had a signifi- 
cant impact on the performance for specific test items, the percent of stu- 
dents within each group responding correctly to each item was compared 
across the three presentation modes. As seen in Table 3, there is only one 
item (#5) for which performance differed significantly (p < .05) across the 
three presentation modes. Two additional items (#2 and 18) show signifi- 
cant differences at the p < .10 level. In all three cases, scores were similar 
between the paper and whole page condition, but were noticeably lower 
for the scrolling condition. 
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Table 3: Comparison of Item Level Performance Across the Three 

Presentation Modes 


Item 

Paper 

Scroll 

Whole 

page 

F 

P 

1 

51% 

53% 

53% 

0.044 

.957 

2 

83% 

67% 

76% 

2.370 

.096 

3 

69% 

54% 

62% 

1.746 

.177 

4 

57% 

56% 

68% 

1.259 

.286 

5 

64% 

41% 

64% 

4.990 

.008 

6 

68% 

59% 

62% 

0.703 

.496 

7 

63% 

49% 

50% 

1.791 

.169 

8 

61% 

53% 

61% 

0.659 

.518 

9 

80% 

79% 

73% 

0.575 

.564 

10 

60% 

64% 

66% 

0.322 

.725 

11 

53% 

59% 

47% 

0.916 

.402 

12 

43% 

47% 

47% 

0.204 

.816 

13 

75% 

64% 

66% 

1.034 

.357 

14 

39% 

37% 

51% 

1.829 

.163 

15 

44% 

49% 

47% 

0.162 

.851 

16 

39% 

31% 

34% 

0.435 

.648 

17 

49% 

51% 

55% 

0.280 

.756 

18 

53% 

37% 

53% 

2.429 

.091 

19 

53% 

41% 

46% 

1.052 

.351 


After reviewing these three items, we could not detect anything 
unusual that might lead to differences in performance across the three 
modes. It is also important to note that since multiple comparisons were 
made across these items, one would expect to find significant difference at 
the .05 level one out of twenty times and two out of twenty times at the 
.10 level hy chance alone. 

As an additional measure of differences in item level scores, an effect 
size statistic (Glass’ delta) was computed for hoth paper/scroll and paper/ 
whole page for each item. Four items (#2, #3, #5, #18) have effect sizes 
greater than 0.3, which is considered hy Cohen (1988) to he in the medium 
effect size range. Again, there does not appear to he anything unusual 
about these items that may lead to these score differences. 
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Examining Score Differences by Prior Computer Skills 
and Use 

To examine the relationship between students’ performance on the 
reading test and their computer fluidity, computer literacy, and prior com- 
puter use, students were divided into three groups based on their scale 
scores. This grouping was conducted separately for the fluidity, literacy, 
home use, and school use scales. For each scale, the three groups repre- 
sented high, medium, and low level fluidity, literacy, or use. For each scale, 
students whose scores were .5 standard deviations above the mean scale 
score were placed into the high group. Students whose scale score was 
within .5 standard deviations of the mean scale score were placed into the 
medium group. And students whose scale scores were .5 standard devia- 
tions below the mean were placed into the low group. After these three 
groups were formed, reading comprehension scores were compared across 
the three presentation modes based on the high-medium-low classifica- 
tion of each computer measure. 

Although students were randomly assigned to the three modes of pre- 
sentation groups, differences in students’ prior computer experiences may 
still exist. If so, these differences may mask differences that result due to 
the presentation mode. To examine differences in prior computer experi- 
ence, mean scores for the fluidity, literacy, home use, and school use scales 
were compared among the three modes of presentation. As seen in Table 4, 
the mean computer fluidity score was highest for the whole page group 
and lowest for the paper group. A similar pattern occurred with com- 
puter literacy, such that the whole page group had higher literacy scores 
(on average) and the paper group had the lowest scores. With respect to 
the home use scale, the whole page group had the lowest scores while the 
scrolling group had the highest scores. Finally, for school use, the scrolling 
group again had the highest school use scale scores while the paper group 
had the lowest scale scores. Despite these visible differences, computer 
fluidity is the only measure for which score differences are statistically sig- 
nificant at the p < .05 level. 

Table 4: Comparison of Scale Scores Across Presentation Modes 


Scale 

Paper 

Mean 

Whole 

Page 

Mean 

Scroll 

Mean 

Pooled 

SD 

F 

Sig. 

Fluidity Scale Score 

-0.19 

0.32 

-0.12 

1.00 

5.195 

.006 

Comp. Literacy (%) 

50.3 

57.5 

54.1 

22.9 

1.780 

.171 

Comp. Literacy Scale Score 

-0.22 

0.16 

0.06 

1.00 

3.014 

.051 

Home Use Scale Score 

0.01 

-0.06 

0.06 

1.00 

0.199 

.820 

School Use Scale Score 

-0.13 

-0.09 

0.24 

1.00 

2.340 

.099 
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To control for differences among the three groups for each of these mea- 
sures, regression analyses were performed. For each regression analysis, 
the scale score was entered as an independent variable and two dummy 
variables were entered to estimate the effect of group membership on 
test scores controlling for differences in scale scores. Specifically, the first 
dummy variable represented the whole page group (i.e., students in the 
whole page group were assigned a 1 and students in the two other groups 
were assigned a 0). The second dummy variable represented the scrolling 
group (i.e., students in the scrolling group were assigned a 1 and students 
in the two other groups were assigned a 0). Students’ reading test score 
was the dependent variable in all the regression analyses. 

The full set of analyses for the computer fluidity, computer literacy, 
home use, and school use scales are presented separately below. 

Computer Fluidity Analysis 

As described above, students were assigned to a high, medium, or low 
computer fluidity groups based on their fluidity scaled score. Table 5 dis- 
plays the average reading comprehension test score for students based on 
their computer fluidity categorization, across each of the three modes. 

Table 5: Comparison of Mean Reading Test Scroll Across Computer Fluidity 

Groups and Presentation Mode 



N 

All 

Students 

Paper 

Scroll 

Whole 

Page 

F 

Sig 

High 

66 

65.8% 

75.0% 

66.8% 

60.2% 

2.429 

.096 

Medium 

66 

56.5% 

59.8% 

50.9% 

58.6% 

1.045 

.358 

Low 

66 

47.8% 

50.4% 

43.2% 

50.4% 

0.764 

.470 


In general, students with higher computer fluidity scores tended to 
score higher on the reading comprehension assessment across all three 
modes. Similarly, students in the low computer fluidity group performed 
lower, on average, on each of the presentation modes. To examine differ- 
ences within each computer fluidity group across presentation modes, a 
one-way ANOVA was run across presentation modes for each fluidity cat- 
egorization. Within each of the three computer fluidity groups, no signifi- 
cant differences were found at the p < .05 level. It is interesting to note, 
however, that for both the medium and the low computer fluidity groups, 
performance in the scrolling condition was lower than the paper or whole 
page conditions. In addition, for both the medium and low computer flu- 
idity groups, performance for the paper and whole page conditions differed 
minimally. Finally, as seen in Figure 3, there appears to be no meaningful 
interaction between fluidity and presentation mode. 
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Figure 3: Mean Reading Score for Fluidity Groups and Presentation Mode 



As described above, a regression analysis was performed in order to 
control for differences in computer fluidity that existed among the presen- 
tation modes. In addition to computer fluidity scale scores, the regression 
model included two dummy variables which represented the scroll and 
whole page group. For this regression model students’ reading compre- 
hension score was the dependent variable. 

As seen in Table 6, the regression model accounted for 14.2% of the 
variance in reading scores. As indicated by the standardized coefficient 
(beta), computer fluidity was the strongest predictor of students’ reading 
scores. Both scrolling and whole page conditions had a negative effect on 
test performance, after controlling for differences in computer fluidity. 
However, the coefficients for both scrolling and whole page were not sta- 
tistically significant at the .05 level. 
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Table 6: Regression Model for Computer Fluidity and Group Membership 

Predicting Reading Scores 


r2 = .i 42 F= 11.88 p<.001 

Variable 

Coefficient 

SE 

Beta 

T ratio 

Sig. 

Intercept 

.609 

.026 


23.054 

<.001 

Fluidity 

.090 

.016 

.387 

5.713 

<.001 

Scroll 

-.069 

.038 

-.139 

-1.832 

.068 

Whole Page 

-.060 

.038 

-.121 

-1.560 

.120 


Computer Literacy Analysis 

As described above, students were assigned to a high, medium, or 
low computer literacy group based on their literacy scaled score. Table 7 
displays the average reading comprehension test score across the three 
presentation modes for students based on their computer literacy catego- 
rization. 

Table 7: Comparison of Mean Reading Test Scroll Across Computer Literacy 

Groups and Presentation Mode 



N 

All 

Students 

Paper 

Scroll 

Whole 

Page 

F 

Sig 

High 

70 

66.4% 

74.4% 

63.0% 

65.3% 

1.564 

.217 

Medium 

71 

58.9% 

62.0% 

56.6% 

57.2% 

0.435 

.649 

Low 

72 

43.4% 

45.6% 

35.1% 

48.4% 

2.495 

.090 


On average, students who scored higher on the computer literacy 
test also scored higher on the reading comprehension test. As depicted in 
Figure 4, this relationship holds across all three presentation modes and 
therefore does not indicate that there is an interaction between presen- 
tation mode and computer literacy. One-way ANOVAs performed within 
each level of computer literacy also indicate that there are no significant 
differences among mean test scores for each presentation mode within 
each computer literacy group. Similar to the fluidity measure, it is note- 
worthy that as students’ computer literacy decreases, the scrolling group 
seems to be more disadvantaged than the whole page group when com- 
pared to the paper group. 
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Figure 4: Mean Reading Score for Computer Literacy Groups and 

Presentation Mode 



As described above, a regression analysis was performed to control 
for differences in computer literacy that existed among the presentation 
modes. As seen in Table 8, the regression model accounted for 21.9% of 
the variance in reading scores. As indicated by the standardized coefficient 
(beta), computer literacy was the strongest predictor of students’ reading 
scores. Both scrolling and whole page conditions had a negative effect on 
test performance, after controlling for differences in computer literacy. 
The coefficient for scrolling was statistically significant at the .05 level. 
This means that after controlling for differences in computer literacy, stu- 
dents who took the test using the scrolling mode achieved a statistically 
significant lower reading comprehension score than students who took 
the assessment on paper and on laptop using whole page text. 
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Table 8: Regression Model for Computer Literacy and Group Membership 

Predicting Reading Scores 


r 2^.219 F = 20.82 p = <.001 

Variable 

Coefficient 

SE 

Beta 

T ratio 

Sig. 

Intercept 

.603 

.024 


24.755 

<.001 

Literacy 

.111 

.014 

.474 

7.696 

<.001 

Scroll 

-.086 

.035 

-.171 

-2.446 

.015 

Whole Page 

-.042 

.035 

-.086 

-1.225 

.222 


Home Use Analysis 

Students’ home-use scale scores were used to divide students into three 
groups which represented high, medium, and low levels of reported use of 
computers at home. Table 9 displays the average reading comprehension 
test score across the three presentation modes for students based on their 
home computer use categorization. 

Table 9: Comparison of Mean Reading Test Scroll Across Home Computer 

Use Groups and Presentation Mode 



N 

All 

Students 

Paper 

Scroll 

Whole 

Page 

F 

Sig 

High 

56 

50.1 % 

51.2% 

51.4% 

47.4% 

0.191 

.827 

Medium 

56 

61 .3% 

64.6% 

51.7% 

63.2% 

1.168 

.319 

Low 

56 

56.6% 

58.6% 

53.8% 

56.8% 

0.188 

.829 


On average, students who were categorized as high home computer 
users scored lower on the reading comprehension test while students who 
were categorized as medium level home computer users scored highest. 
As depicted in Figure 5, however, this relationship holds for the paper and 
whole page presentation modes, but does not hold for the scroll mode. 
Specifically, there are only minor visible differences in performance 
between the three home use groups for the scrolling condition. One-way 
ANOVAs performed within each level of home computer use indicate that 
there are no significant differences among mean test scores for each pre- 
sentation mode within each home computer use group. 
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Figure 5: Mean Reading Score for Home Use Groups and Presentation Mode 



Although there were only small differences in mean home computer 
use scale scores among the three presentation modes, a regression model 
was developed to examine the effect of presentation mode controlling for 
differences in home computer use. The model accounted for less than 3% 
of the variance in reading comprehension and neither the whole page nor 
the scrolling presentation modes were significant predictors of reading 
scores. 
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School Use Analysis 

Students’ school-use scale scores were used to divide students into 
three groups which represented high, medium, and low levels of reported 
use of computers in school. Table 10 displays the average reading compre- 
hension test score across the three presentation modes for students based 
on their school computer use categorization. 

Table 1 0: Comparison of Mean Reading Test Scroll Across School Computer 

Use Groups and Presentation Mode 



N 

All 

Students 

Paper 

Scroll 

Whole 

Page 

F 

Sig 

High 

57 

49.0% 

54.9% 

47.7% 

46.8% 

0.877 

.422 

Medium 

57 

61.5% 

58.2% 

63.6% 

63.4% 

0.325 

.724 

Low 

57 

56.9% 

54.3% 

55.8% 

59.7% 

0.297 

.744 


As seen in Table 10 and in Figure 6, there is not a linear relationship 
between students reported school computer use and reading comprehen- 
sion test scores. Within the paper group, reading scores differ only slightly 
between the high, medium, and low school-use groups. For the scrolling 
and whole page groups, the medium-level school use group performed 
noticeably higher than did the high use group. It is difficult to hypothesize 
why this pattern occurs. Within each level of school computer use, a one- 
way AN OVA indicates that there are not statistically significant differences 
in reading scores across the mode of presentation. 
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Figure 6: Mean Reading Score for School Use Groups and Presentation Mode 



A regression analysis was performed to control for differences in school 
computer use that existed among the presentation modes. The model 
accounts for less than 2% of the variance in reading comprehension scores 
and neither the whole page nor scrolling conditions were significant pre- 
dictors of reading scores. 
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Gender Analysis 

To examine whether the effect of presentation mode differed for hoys 
and girls, reading test performance was compared between gender within 
and across each presentation mode. As seen in Table 11, overall girls 
received higher reading, computer literacy, and school use scale scores than 
did boys. Conversely, boys received higher computer fluidity and home use 
score. Overall, however, these differences were small and not statistically 
significant. 

Table 1 1 : Gender Comparison Across All Measures and Presentation Modes 



Reading 

Compre- 

hension 

Computer 

Literacy 

Fluidity 

Home Use 

School 

Use 

All 

Boys (112) 

53.9% 

52.9% 

0.102 

0.056 

-0.075 

Girls (106) 

58.2% 

55.1% 

-0.101 

-0.057 

0.091 

Paper 

Boys (37) 

53.9% 

46.0% 

-0.22 

0.034 

-0.311 

Girls (37) 

63.3% 

54.5% 

-0.16 

-0.005 

0.097 

Scroll 

Boys (34) 

54.3% 

57.2% 

0.17 

0.142 

0.317 

Girls (36) 

50.1% 

51.1% 

-0.35 

-0.020 

0.162 

Whole 

Page 

Boys (41) 

53.4% 

55.5% 

0.35 

0.011 

-0.182 

Girls (33) 

61 .2% 

59.8% 

0.28 

-0.141 

0.023 


However, as seen in Figure 7, an interaction effect may occur between 
gender and the mode of presentation effect. As seen in Figure 7, boys per- 
form about the same across all three conditions. In addition, girls perform 
noticeably better than boys under the paper and whole page conditions. 
For the scroll condition, however, girls perform noticeably worse than 
they do under the two other conditions and also perform slightly worse 
than boys. 
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Figure 7: Mean Reading Score for Gender and Presentation Mode 



Presentation Mode 

Exploratory Analysis of lEP and Multi-Ungual Students 

After the data collection phase of this research was completed, Boston 
College researchers asked the participating teachers and principals to 
provide information about whether students in their classrooms had 
language-related lEPs (Individualized Education Plans). One teacher did 
not respond to the request and one principal responded that none of the 
students in her school had an lEP The remaining teachers and principals 
provided the names of 25 students who had language-related lEPs. Based 
on this information, all students were coded as either “lEP” or “Non-IEP.” 
Overall, lEP students scored more than 20% points lower than non-IEP 
students on the reading comprehension assessment and also scored lower 
on the fluidity exercises and literacy assessment. These results are not sur- 
prising, since the lEP students have heen identified as having language- 
related difficulties. We next examined the relationship between the lEP 
students’ performance on the reading comprehension assessment and the 
mode of delivery. Table 12 shows that lEP students scored similarly across 
the three modes of delivery. In other words, the computer-based testing 
modes neither advantaged nor disadvantaged the lEP students. 
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Table 1 2 Comparison of Reading Scores Based on lEP Status 



Reading Comprehension Assessment 

Paper 

Whole Page 

Scrolling 

lEP 

36 . 3 % 

37 . 4 % 

34 . 2 % 

Non-IEP 

61 . 5 % 

59 . 6 % 

53 . 9 % 


As seen in Figure 8, there does not appear to be any interactions 
between lEP status and performance on each passage presentation condi- 
tion. 

Figure 8: Mean Reading Score for lEP Groups and Presentation Mode 
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Open-Ended Survey Items 

The above analyses provide insight into the extent to which perfor- 
mance differed across the three presentation modes and how prior com- 
puter uses and experiences may have affected test performance within 
each condition. To collect information from the students’ perspective 
on their experiences performing the reading test on computer, students 
in the scrolling and whole page groups were asked to responded to four 
open-ended survey items. A content analysis was performed on students’ 
responses to these four items. Patterns that emerged from this content 
analysis are summarized for each open-ended item. 

1. Overall, was taking the test on a computer easier or more difficult 
than taking the test with paper and pencil? Why? 

Of the 135 students who responded to this item, 82.2% reported that 
they thought it was easier to take the test on computer. The most common 
reason provided for why the test was easier on computer was that students 
did not have to write/their hand did not get tired from writing (this was 
sited even though students were responding to multiple-choice items). 

2. Describe any problems that you had when taking the test on the 
computer. 

Twenty-six students indicated that they had a problem while taking the test 
on computer. The most frequently cited problems were not understanding 
the directions, having problems using the mouse, not understanding how 
to end the test, and having the computer freeze. 

3. When reading the passage, did you underline or highlight any text? 
Why or why not? 

Of the 137 students who responded to this item, 16.8% reported using the 
highlighting feature. The most common reason for using highlighting was 
to remember important information and the most common reason for not 
using the highlighting was because they felt they did not need to use this 
feature. 

4. When taking the test, did you mark any questions that you were 
unsure about and wanted to work on later? Why or why not? 

Of the 132 students who responded to this item, 17.4% said that they did 
mark items for review. The most common reported reason for marking 
for review was because they wanted to go back to it later and the most 
common reason for not marking for review was because they were confi- 
dent in their answer or felt that they did not need to use this feature. 
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Discussion 

The pilot study presented here was limited to 219 fourth grade students 
from one state who performed a reading comprehension test of moderate 
length. Findings from a pilot study should not be interpreted as definitive. 
With this caution in mind, the results highlight some interesting findings 
and suggest that more research on a larger scale should be conducted in 
the area of computer-based delivery of passage dependent assessments. 
Below, we summarize the major findings. 

1. There were no significant differences in reading comprehension scores 
across testing modes. 

On average, students in the paper group (n-75) answered 58.1% of the 
items correctly, students in the scrolling group (n-70) answered 52.2% of 
the items correctly, and students in the whole page group (n=74) answered 
56.9% of the items correctly. Although there is almost a 6% point differ- 
ence in scores between the paper and scrolling groups, this difference was 
not significant at the p < .05 level. 

Without examining other student measures, finding no significant dif- 
ferences in test scores across paper and computer delivered modes implies 
that overall students were neither advantaged nor disadvantaged by the 
mode of test delivery. However, characteristics of the sample and differ- 
ences that may occur for students with different levels of prior computer 
experience must also be considered. 

Although the sample of schools that participated in this study repre- 
sent a range of schools within Vermont, the schools and their students 
were not randomly selected. Rather they were recruited based on their 
location, composition of their student body, and, perhaps most impor- 
tantly, their willingness to participate. As part of the recruitment process, 
principals received recruitment letters that detailed the research ques- 
tions and described the data collection procedures. Although this recruit- 
ment process was necessary in order to obtain school-level support for the 
data collection, this process may have resulted in the selection of schools 
in which principals were more interested in the use of technology or in 
computer-based testing specifically. Although efforts were made to attain 
a representative sample of Vermont 4* grade students, sample bias is a 
concern that limits the generalizability of the results. 

As a measure of socio-economic status, students were asked the fol- 
lowing question on the survey: “About how many books of your own do 
you have at home, not counting your school books or comic books?” The 
response options were: a) 5 or fewer b) 6-25 c) 26-50 d) 51-100 e) more 
than 100. Only nine students reported that they have 5 or fewer books 
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of their own at home and the most common response (n=86, 39.3%) was 
“more than 100.” Students were also asked to report how many computers 
they have at home. The response options were a) 0 h) 1 c) 2 d) 3 or more. 
Again, only nine students reported they do not have a computer at home. 
The most common response (n=105, 48.6%) was 1, while 54 (24.7%) stu- 
dents reported that they have 2 computers at home, and 48 (21.9%)stu- 
dents reported that they have 3 or more computers at home. Students’ 
answers to these two items indicate that the sample of 219 students does 
not include many students from a low socio-economic and low computer 
access population. Although the results suggest that, across all students, 
the modal effect is not statistically significant, this finding may be due 
in part to the atypically high computer access and higher socio-economic 
status of the sample. 

2. There were no statistical differences in reading comprehension scores 
based on computer fluidity and computer literacy, but a pattern in 
performance may indicate that students are disadvantaged by the 
scrolling text mode, particularly students with lower computer skills. 

Computer-based testing experts hypothesize that scrolling may nega- 
tively impact students’ test scores in passage based assessments (Choi 8t 
Tinkler, 2002, Pommerich, 2004). Students who are not as familiar with 
using a computer may have difficulty navigating through a passage for 
which they are required to use the scroll bar. Although this hypothesis was 
not supported in a statistically significant manner, there is evidence of 
modal differences in student performance based on their computer skills 
and knowledge. 

As seen in Table 5, students’ reading comprehension scores decrease 
as their computer fluidity decreases. For all three fluidity groups, scores 
are noticeably lower for the scrolling mode than on the paper version. For 
both the medium- and low-level fluidity groups, the lowest scores occur 
on the scrolling version (see Figure 3). Similarly, for all three computer 
literacy groups, scores are lower for the scrolling mode than on the paper 
version. For the lowest computer literacy group, this difference is substan- 
tial (Figure 4). This pattern may indicate that students generally perform 
worse under the scrolling condition than under the paper or whole page 
conditions and that the score differences are largest for students with lower 
computer fluidity and/ or literacy. Again, while these patterns are not sta- 
tistically significant, they warrant further investigation with a larger and 
more diverse sample of students. 
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3. The majority of students who took the reading test on a computer 
indicated that they would prefer to take the test on computer. 

After completing the selected response portion of the survey, stu- 
dents who completed the assessment on computer were asked to answer 
four open-ended questions. One question asked students whether they 
thought it was easier or harder to take the test on computer or paper. Of 
the 135 students who responded to this item, 82% reported that it was 
easier to take the test on computer. In addition, students were asked in a 
selected response format if they would have preferred to take the test on 
computer or on paper. Of the 161 students who responded to this item, 
87% reported that they would prefer to take the test on computer. In addi- 
tion, when asked about problems, very few students noted that they expe- 
rienced difficulty taking the test on computer. 

Completion rates also provide evidence of students’ ability to perform 
the reading test on the computer. Only nine students did not reach the 
last item on the reading comprehension assessment. These nine students 
were evenly distributed across the paper, scrolling, and whole page groups. 
Also, the number of students who answered all of the items was higher for 
the computer groups than for the paper group. Although this sample did 
not include many students who had limited prior computer experience, 
the survey responses, completion rates, and student observations are all 
evidence that computer anxiety generally did not interfere with students’ 
ability to take the assessment. 

4. Providing highlighters and review markers is useful for some 
students. 

One concern raised when transitioning a test from paper to com- 
puter focuses on the belief that examinees may be disadvantaged when 
features such as highlighting and being able to skip and return to items 
are not available. Choi and Tinkler (2002) hypothesized that “providing 
such accommodations as an electronic marker, which allows students to 
highlight a selection of text in a passage, will not only simulate the nat- 
ural pencil-and-paper test-taking practice, but also reduce the administra- 
tion mode effect to a certain extent” (p. 10). In an effort to create similar 
testing environments in the paper and computer groups, highlight and 
mark for review tools were built into the computer-based test interface for 
this research. Through a survey administered after the completion of the 
test, less than 20% of students indicated that they used these features. 
However, based on open-ended survey responses, it appears that those 
students who used the features did so in a manner that was consistent 
with the intended purposes. 
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Students who reported that they used the highlight feature listed rea- 
sons such as rememhering information, noting important parts of the 
passage, and wanting to go hack to the highlighted portions as reasons for 
using this feature. The majority of students who listed reasons for using 
the mark for review feature reported that they were unsure of the answer 
to an item and wanted to return to the item later. These appropriate rea- 
sons for highlighting text and marking items for review suggest that a 
small proportion of students may benefit from having access to these test 
taking tools in the computer-based environment. However, more than 20 
students who reported that they did not use the highlighter and 8 students 
who reported that they did not use the mark for review feature noted that 
they either did not understand how to use the tools or forgot to use them. 
This finding indicates that further training and practice may have resulted 
in higher usage of the highlighter and mark for review tools. 


Conclusion 

In this study of the impact of transitioning from paper to two com- 
puter-based testing formats for passage based reading comprehension 
assessments, no significant differences in test scores were found between 
the three groups. Taking multiple measures of students’ computer skills 
allowed rich data analysis to understand the relationship between com- 
puter use and performance on computer-based assessments. Although the 
size and scope of this study were not large enough to show definitively that 
scrolling negatively affects students with low computer skills and knowl- 
edge, the results suggest that further research is warranted to more fully 
understand the impact of scrolling on passage-based assessments. 

Future research should be conducted on a larger and more diverse 
sample of students and should be expanded to include students in other 
grade levels. The number of passages and items should also be increased. 
The high completion rate and observation of some students finishing the 
test in approximately 20 minutes is evidence that the test length could be 
increased. Adding more well constructed items would increase test reli- 
ability and provide a better opportunity to examine if there are differences 
in completion rate across test modes. This future study with a larger and 
more diverse sample and more items may show more subtle differences in 
modal performance overall and in student sub-groups. This research could 
lend further insight into the use of computer-based test features, and 
reveal relationships between student performance and their computer 
skills, knowledge, or use. 
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Endnote 

1 To ensure that the scores of the three paper students who missed items on the 
hack of pages did not affect the overall research results, a sensitivity analysis was 
conducted which assumed that the students would have answered the missed items 
correctly. This analysis showed no changes in the study finding of non-significant 
differences in performance across the three testing modes. 
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